Skip to content

fix: preserve completion usage and trim stop tokens - #312

Draft
Andrewxu313 wants to merge 1 commit into
mainfrom
tairan/glm52-completion-fix
Draft

fix: preserve completion usage and trim stop tokens#312
Andrewxu313 wants to merge 1 commit into
mainfrom
tairan/glm52-completion-fix

Conversation

@Andrewxu313

Copy link
Copy Markdown
Contributor

Description

Preserve OpenAI prompt-token usage across sequence re-entry and trim model stop tokens consistently from completion text.

Motivation

Sequence re-entry mutates the internal prompt length while recomputing KV state, but response usage must retain the caller's original prompt. Two completion detokenization paths also bypassed the existing EOS-aware helper and could leak stop tokens.

Milestone (optional)

GLM-5.2 release closeout.

Type of Change

  • model — add/extend model support (models/** + registration seam + model kernels only)
  • kernel — add/optimize a compute kernel (batchgen_kernels/** + in-tree kernel dirs)
  • core — change scheduling/serving/runtime scaffolding (the only type that may)
  • fix — narrow bug fix (+ a regression test)
  • infra — build / CI / packaging / scripts / Docker
  • docs — documentation only

File changes

File Δ Note
batchgen/batchgen_worker.py mod Use original prompt usage and shared EOS-aware detokenization.
tests/test_completion_usage_accounting.py add Cover re-entry usage plus gathered/fallback stop trimming.

Validation

  • PR hygiene: PASS for PR_TYPE=fix.
  • Focused remote tests passed on the aggregate release branch containing the identical patch.
  • Clean reconstructed-branch remote gate: pending before ready-for-review.

Checklist

  • I have read the CONTRIBUTING guide and PR Merge Policy Contract.
  • I have updated the tests.
  • I have updated the documentation if applicable.

PR Merge Policy Contract — pre-merge checklist

  • Diff reviewed; every file traces to the task.
  • Exactly one type is selected and all files fit it.
  • File table matches the diff.
  • No scratch/debug production scripts.
  • Tests are under tests/.
  • No new environment-variable debug guard.
  • No leftover debug output or commented-out code.
  • No artifacts staged.
  • No public API change requiring module docs.
  • One surgical concern.
  • Commit format and body comply with policy.
  • CI is green.

Use the original prompt length for response usage after sequence re-entry, and route completion detokenization through the shared EOS-trimming helper.
@github-actions github-actions Bot added the ci:run Trigger build + GPU regression on H20 label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:run Trigger build + GPU regression on H20

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant